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1.  Introduction 


The  flow  of  accurate  information  and  proper  situational  awareness  can  make  the 
difference  between  a  successful  mission  and  a  failed  mission.  The  battlefield  that 
our  Soldiers  are  on  is  dynamic;  thus,  Soldiers  must  adapt  their  information¬ 
gathering  techniques  and  fighting  styles  to  tailor  to  the  constantly  changing 
battlefield.  A  new  type  of  warfare  focus  has  been  developed  in  the  age  of 
information  to  help  Soldiers  adapt.  This  warfare  focus  is  called  Network  Centric 
Warfare  (NCW).  Essentially,  NCW  translates  increased  information  gathering  and 
information  superiority  into  increased  combat  power  superiority  on  the  battlefield. 
NCW  allows  commanders  to  get  the  most  out  of  their  people  and  assets  (Alberts  et 
al.  2000). 

Great  importance  lies  in  the  ability  of  commanders  to  present  information  in  the 
most  effective  way  possible.  Too  much  information  can  cause  a  cognitive  overload, 
but  enough  information  is  needed  to  provide  situational  awareness  for  effective 
decision  making.  This  report  overviews  a  preliminary  investigation  simulating  an 
NCW  environment  in  the  form  of  a  simple,  yet  challenging,  war  game.  The  game, 
titled  Shadow  Force,  requires  participants  to  make  decisions  based  on  how  the 
information  is  visually  presented.  The  results  from  the  simulated  game  allow  us  to 
determine  the  most  effective  way  to  display  gathered  information  to  increase 
information  superiority,  and  thus  combat  superiority,  through  NCW. 

Shadow  Force  was  based  on  a  game  titled  Scud  Hunt.  Scud  Hunt  was  a  digital  game 
originally  developed  by  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  and  ThoughtFink,  Inc.  to  determine  how  different  communication  means 
and  visual  displays  created  a  shared  situational  awareness  for  a  mission  command 
scenario  within  a  team  (Perla  et  al.  2000).  Scud  Hunt  used  a  5  x  5  square  grid  to 
represent  the  battlefield.  Teams  of  4  participants  were  given  different  types  of  assets 
to  complete  their  mission  of  finding  4  hidden  Scud  missile-launching  devices  in  a 
certain  number  of  rounds.  The  different  assets  had  varying  degrees  of  reliability 
and  did  not  always  provide  accurate  information.  The  teams  had  to  use  their  assets 
effectively  and  communicate  efficiently  to  complete  the  mission  of  finding  all  4 
launchers  within  a  specific  number  of  rounds.  Shadow  Force  uses  the  same  play 
style  and  concept  as  Scud  Hunt  to  test  a  variety  of  performance  variables. 

2.  Experimental  Design 

This  section  describes  participants  for  the  experiment,  how  Shadow  Force  operates, 
and  what  data  was  analyzed.  Prior  experimental  iterations  of  Shadow  Force  were 
run  to  test  different  variables,  including  different  numbers  of  rounds,  different 
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amounts  of  time  to  complete  the  mission,  and  varying  levels  of  information  being 
presented  (Jackson  and  Cassenti  2017).  The  main  focus  of  the  current  Shadow 
Force  experiment  was  to  see  if  a  visual  map-based  display  using  symbols  caused  a 
difference  in  results  compared  to  a  text -based  situation  report. 

Participants  were  randomly  assigned  either  a  map-based  grid  with  symbols  or  a 
text-based  report  that  would  confirm  the  presence  of  an  enemy  launcher,  give  the 
possibility  of  a  launcher  being  present,  or  deny  that  an  enemy  launcher  was  present 
through  the  use  of  symbols  or  text.  The  key  research  question  was:  Which 
information  format  would  generate  the  highest  average  number  of  launcher  hits? 
The  number  of  hits  was  used  as  a  measure  of  success  to  determine  which 
information  format  was  the  most  efficient. 

Approximately  100  participants  were  recruited  online  from  the  Amazon 
Mechanical  Turk  service  for  the  Shadow  Force  experiment.  Two  different 
information  display  methods  were  used  to  test  which  method  was  more  efficient. 
Half  of  the  participants  used  only  a  text-based  information  display  that  described  if 
an  asset  identified  no  enemy  present,  a  possible  enemy  present,  or  a  confirmed 
enemy  present  for  each  round  (represented  as  one  day)  of  the  game.  The  other  half 
of  the  participants  used  a  map-based  information  display  showing  corresponding 
symbols  for  no  enemy,  possible  enemy,  and  confirmed  enemy  on  their  map.  Figure 
1  shows  the  map-based  display  using  symbols.  Figure  2  shows  the  text-based 
display  using  a  situation  report. 


(T)[  Current  | 
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1 

4 

1 

1 

i 
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Fig.  1  Screen  capture  of  the  map-based  information  display  using  symbols 
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Situation  Reports 


Day  1  of  5: 

Siglnt  Possible  target  at  B2 

Spy  No  target  at  B5 

Spy  Dead 

Spec  Ops  Target  confirmed  at  E5 

Navy  SEALS  No  target  at  E4 

Recon  Satellite  No  target  at  A3.B3.C3.E3 

Recon  Satellite  Possible  target  at  D3 

Manned  Aircraft  No  target  at  Cl  ,C2,C3,C4 

Manned  Aircraft  Target  confirmed  at  C5 

Manned  Aircraft  Will  be  ready  next  turn 

UAV  No  target  at  D1.D2.D3  D4  D5 

Day  2  of  5: 


Fig.  2  Screen  capture  of  the  text-based  display  using  a  situation  report 


2.1  Gameplay 

Shadow  Force’s  gameplay  assigns  2  human  participants  a  set  of  assets.  There  are  2 
different  sets  of  assets:  air  assets  (Satellite,  Manned  Aircraft,  and  UAV)  and  ground 
assets  (Siglnt,  Spy,  Spec  Ops,  and  Navy  Seals).  Participants  are  instructed  to  place 
their  assets  within  a  5  x  5  grid  with  the  knowledge  that  each  asset  performs  at  a 
different  level  of  accuracy  and  reliability.  For  example,  the  Spec  Ops  team  asset 
can  be  deployed  to  any  one  square  and  can  only  move  one  square  per  turn,  but  is 
very  reliable  in  identifying  missile  launchers.  On  the  other  hand,  the  Satellite  asset 
can  scan  an  entire  column,  but  cannot  reliably  confirm  the  presence  of  a  missile 
launcher.  Each  participant  is  able  to  receive  information  from  their  assets  at  the  end 
of  the  round,  including  the  other  participant’s  assets.  After  each  round,  the 
information  gathered  will  either  be  displayed  in  text  form  on  the  situation  report,  or 
in  symbol  form  on  the  grid,  depending  on  which  display  method  the  participant  was 
assigned.  The  participants  have  5  rounds  (represented  as  days)  to  find  4  missile 
launchers.  Each  round  lasts  150  s.  Participants  must  use  their  assets  in  coordination 
with  each  other  to  find  all  of  the  launchers.  For  more  detailed  information  on  the 
instructions  for  Shadow  Force,  see  Jackson  and  Cassenti  (2017). 

The  gameplay  begins  with  a  brief  scenario  and  background  for  participants: 

The  rogue  state  of  Porona  has  acquired  mobile  ballistic  missiles  and 
weapons  of  mass  destruction.  Porona  is  threatening  our  ally,  Kartuna, 
located  across  the  narrow  Gulf  of  Sabani.  Your  mission  is  to  locate  the 
missile  launchers,  using  various  ground,  space,  air,  and  intelligence  assets. 

The  elite  fanatical  Poronan  Revolutionary  Guard  Special  Artillery 
Regiment  (PRGSAR),  with  a  number  of  mobile  missile  launchers,  has 
deployed  from  its  depot  to  a  secret  hide  site.  This  deployment  is  supported 
by  deception  operations  that  may  confuse  our  assets.  You  must  identify 
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the  locations  of  4  mobile  missile  launchers  in  the  next  5  days.  The  fate  of 
hundreds  of  thousands  of  Kartunans  depends  on  your  success. 

Next,  the  participants  play  in  a  tutorial  round  where  they  learn  the  basics  of  the 
game,  as  show  in  Fig.  3.  The  tutorial  gives  the  participants  a  basic  overview  of  how 
to  play  and  allows  the  participants  to  try  different  assets.  In  the  tutorial,  the 
participants  see  both  the  map-based  display  and  the  text-based  display.  The 
participants  play  through  5  rounds  with  gameplay  tips  appearing  at  the  top  of  the 
screen.  For  the  tutorial,  the  participants  are  playing  by  themselves,  but  for  the  actual 
game  they  are  paired  with  a  human  partner  who  has  a  different  set  of  assets. 


You're  playing:  Shadow  Force 


Your  Assets 


Situation  Reports 


Recon  Satellite 

Scan  column 

E 

G 

Siglnt 

E 

& 

spy 

want  (ram  mere 

Recon  Satellite 
Recon  Satellite 
Recon  Satellite 
Siglnt 

spy 

Manned  Aircraft 

Manned  Aircraft 

UAV 

UAV 

UAV 

Spec  Opt 
Navy  SEALS 

|Pay  2  of  5: 


No  target  at  E3 
Possible  target  at  03 
Target  confirmed  at  A3  B3.C3 
No  target  at  83 
No  target  at  05 
No  target  at  El  E2  E3.Ea.ES 
■  be  ready  next  turn 
no  target  at  Cl  C4.CS 
Target  confirmed  at  C2.C3 
WM  be  ready  in  2  turns 
Target  confirmed  at  AS 
No  target  at  E 2 


Fig.  3  A  screen  capture  of  the  tutorial 

After  the  tutorial,  the  participants  are  placed  into  a  practice  game  with  a  computer 
partner  (a  bot).  This  practice  round  will  randomly  assign  them  a  set  of  assets  (air  or 
ground)  and  one  of  the  display  formats  (map-based  or  text-based).  Figure  4  is  a 
screen  capture  of  the  active  map-based  display  condition  during  gameplay.  In  this 
condition,  the  situation  report  only  tracks  which  round  (day)  the  participants  are  on. 
The  map  grid  displays  the  symbols  corresponding  to  the  information  gathered  from 
both  players’  assets  from  previous  rounds.  Figure  5  is  a  screen  capture  of  the  text- 
based  display  condition  during  gameplay.  In  this  condition,  the  situation  report 
displays  all  of  the  asset  information  gathered  from  the  previous  rounds.  The  map 
only  displays  where  the  participants  are  placing  their  assets  for  the  current  round. 
The  display  method  used  in  the  practice  game  determines  the  display  method  used 
in  the  actual  game. 


Approved  for  public  release;  distribution  is  unlimited. 

4 


Shadow  Force 


1:20 

flj[  Current  | 

1  2  3  4  5 


End  Turn 


Markers:  [~j~|  (  ]  f0~]  [~C~] 


Your  Assets  Situation  Reports 


Fig.  4  A  screen  capture  of  the  map-based  display  method 


Shadow  Force 


1:19  | 


Your  Assets  Situation  Reports 


+  Recon  Satellite 

Scan  column 

Day  1  of  5: 

Recon  Satellite 

c 

No  target  at  B4.C4.E4 

Possible  target  at  D4 

Target  confirmed  at  A4 

Manned  Aircraft 

Recon  Satellite 

< — '  _y  Scan  a  row  up  to  3  Resting 

Manned  Aircraft  No  target  at  E2,E3,E4,E5 

squares  in  from  the 

Q 

Manned  Aircraft  Target  confirmed  at  El 

coast 

Manned  Aircraft  Will  be  ready  next  turn 

UAV 

1 

UAV 

No  target  at  D1.D2.D3  1 

Scan  Row 

UAV 

Possible  target  at  D4 

H 

UAV 

Target  confirmed  at  D5  1 

Siglnt 

Possible  target  at  C2 

Spy 

No  target  at  E5 

Spy 

Dead 

Spec  Ops 

Possible  target  at  E5 

Fig.  5  A  screen  capture  of  the  text-based  display  method 


In  the  map-based  display,  the  different  symbols  appear  on  the  grid  in  the  respective 
squares  that  were  inspected  by  an  asset  for  each  round.  The  “0”  represents  no  target 
found,  the  “?”  represents  a  possible  target,  and  the  “!”  represents  confirmed  target. 
This  information  is  updated  at  the  end  of  every  round.  The  symbols  from  previous 
rounds  stay  on  the  map  until  the  game  is  over. 
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In  the  text-based  display,  each  asset  gives  a  description  of  what  was  found  in  the 
squares  they  searched  (e.g.,  “No  target  at  E3”,  “Possible  target  at  D4”,  and  “Target 
confirmed  at  A4”).  The  situation  report  is  updated  at  the  end  of  each  round  under 
the  day  that  information  was  gathered.  This  information  stays  in  the  situation  report 
until  the  game  is  over. 

After  the  practice  round,  the  participant  is  placed  in  the  actual  game  and  paired  with 
another  human  participant  who  has  also  completed  the  tutorial  and  practice  rounds. 
Both  of  the  participants  are  assigned  a  set  of  assets  (air  or  ground)  and  either  a  map- 
based  or  text-based  display.  Both  of  the  participants  use  the  same  information 
display  (e.g.,  both  participants  will  have  a  map  display  or  both  will  have  a  text 
display). 

The  actual  game  plays  exactly  like  the  practice  round.  After  all  of  the  5  rounds  have 
concluded  the  participants  have  150  s  to  select  where  they  think  the  4  missile 
launchers  are  located.  The  participants  have  all  of  the  information,  including  their 
partners’  information,  from  all  5  rounds.  Participants  physically  place  launchers 
where  they  believe  the  actual  launchers  are  located  in  the  5x5  grid.  Figure  6  shows 
a  participant  placing  their  launchers. 


Shadow  Force 


0:56 


You're  playing:  Shadow  Force 


FlFlFlFIFll  current  | 


Target  Locations  Situation  Reports 


Site  1 

mobile  missile  launcher  ^ 


Site  2 

mobile  missile  launcher 


% 


Site  3 

mobile  missile  launcher  C4 


Site  4 

mobile  missile  launcher  52 


Day  1  of  5: 
Day  2  of  5: 
Day  3  of  5: 
Day  4  of  5: 
Day  5  of  5: 


Fig.  6  A  screen  capture  of  a  participant  placing  their  launchers 


Approved  for  public  release;  distribution  is  unlimited. 

6 


Once  the  participant  has  placed  the  launchers,  they  are  asked  to  rate  their 
confidence  of  correctly  placed  launchers  based  on  a  Likert-type  scale,  with 
responses  ranging  from  “Very  unsure”  to  “Very  confident”,  as  seen  in  Fig.  7. 

You're  playing:  Shadow  Force 

How  confident  are  you  that  you  found  the  targets? 


0  di®  © 

Fig.  7  A  screen  capture  of  the  participant  selecting  their  confidence  level 

2.2  Post-Game  Survey 

The  post-game  survey  allows  users  to  reflect  on  their  experience  in  the  game. 
Questions  include: 

1.  What  was  your  strategy? 

2.  Did  you  write  down  any  information  reported  from  the  assets  for  later 
reference? 

3.  Rank  the  assets  in  the  order  of  most  relied  upon. 

4.  Rank  the  assets  in  the  order  of  most  trustworthy. 

For  question  1,  the  participants  are  given  a  textbox  to  freely  write  how  they  played 
through  the  game.  For  question  2,  the  participants  are  given  a  yes  or  no  option.  For 
question  3,  the  participants  are  allowed  to  drag  and  drop  the  different  assets  into  a 
list.  The  asset  that  the  participant  relied  on  the  most  is  put  at  the  top  and  the  asset 
the  participant  relied  on  the  least  is  placed  at  the  bottom.  For  question  4,  the  format 
is  the  same  as  question  3,  but  the  participants  are  told  to  drag  and  drop  the  assets 
that  they  believe  are  the  most  trustworthy.  The  most  trustworthy  is  put  at  the  top, 
and  the  least  trustworthy  is  put  at  the  bottom. 

The  results  from  this  survey  allow  us  to  evaluate  how  the  participants  approached 
the  game,  and  what  type  of  assets  they  liked  the  best. 
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3.  Results 


The  results  from  the  100  participants  playing  Shadow  Force  allowed  us  to  see  which 
information  display  was  the  most  effective  in  improving  the  participant’s 
situational  awareness  to  complete  the  mission. 


3.1  Analysis 


Half  of  the  participants  (50)  were  randomly  assigned  to  the  text-based  situation 
report  and  the  other  half  were  assigned  to  the  map-based  symbol  display.  There  was 
a  significant  difference  between  the  text  display  and  map  display  in  the  number  of 
launchers  hit:  t  (198)  =  5.95,  p  <  0.01,  with  the  map  display  hitting  more  launchers 
on  average.  As  seen  in  Fig.  8,  the  participants  using  the  map-based  symbols  scored 
higher  (M  =  2.35,  SD  =  1.04)  than  the  participants  using  the  text-based  situation 
report  (M  =  1.48,  SD  =  1.03)  in  terms  of  missile  launchers  hit. 


Average  Number  of  Hits  by  Display  Format 


2.5 


Map-Based  Text-Based 


Condition 


Fig.  8  Shows  the  average  number  of  launcher  hits  for  each  display  method 

While  the  map-based  display  produced  better  results  in  terms  of  targets  hit,  the  level 
of  confidence  recorded  by  the  Likert-type  scale  showed  that  participants  using  the 
text-based  situation  report  were  more  confident  in  their  selection  of  possible  missile 
launcher  locations.  This  statistic  is  surprising,  since  the  map-based  participants 
generally  performed  better  than  the  text-based  participants.  There  is  a  significant 
difference  between  the  map-based  confidence  and  the  text-based  confidence: 
t  (198)  =  1.23 ,p  <  0.05.  As  seen  in  Fig.  9,  the  text-based  confidence  (M  =  2.44,  SD 
=  1.03)  was  higher  on  average  than  the  map-based  confidence  (M  =  2.26,  SD  = 
0.95). 
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Average  Confidence  by  Display  Format 

2.5 


2.15 


Map-Based  Text-Based 

Condition 


■  Total 


Fig.  9  Shows  the  average  confidence  based  on  which  information  display  the  participant 
was  assigned. 

While  analyzing  the  confidence  levels  for  each  of  the  conditions,  an  interesting 
correlation  emerged  from  comparing  the  number  of  launchers  hit  and  the 
participant’s  confidence  in  their  choice.  The  participants  who  were  more  confident 
in  their  placement  of  the  launchers  actually  hit  fewer  launchers  than  the  participants 
who  were  less  confident.  For  the  participants  using  the  text  display,  the  correlation 
coefficient  was  -0.158.  This  means  that,  as  their  confidence  increased,  the 
participants  hit  fewer  missile  launchers.  The  map-based  participants  produced 
similar  results  with  a  coefficient  of  -0.042.  Figure  10  combines  the  confidence  of 
both  map-based  and  text-based  participants  into  a  scatterplot  to  show  total 
participants’  confidence  decreased  as  hits  increased.  The  dots  on  the  scatterplot 
represent  the  participants  that  got  those  corresponding  hits  and  confidences. 
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Fig.  10  Shows  the  declining  number  of  hits  for  increasing  confidences  for  both  conditions 
combined 

3.2  Discussion 

On  today’s  battlefield,  information  is  power.  NCW  says  that  increased  information 
superiority  will  translate  into  increased  combat  superiority.  Our  Soldiers  need  to  be 
able  to  accurately  and  quickly  identify  the  information  being  presented  to  them. 
High-quality  information  will  enhance  a  Soldier’s  situational  awareness  and  will 
result  in  being  able  to  complete  the  mission  more  effectively. 

When  comparing  the  2  different  displays,  the  participants  with  the  map-based 
display  had  better  situational  awareness.  They  were  able  to  complete  the  mission 
(locating  missile  launchers)  more  effectively.  The  participants  using  the  text-based 
display  were  able  to  complete  the  mission,  but  not  as  accurately  as  the  participants 
using  the  map  display.  Information  overload  could  have  played  a  role  in  why  the 
text-based  participants  scored  lower  than  the  map-based  participants.  The  situation 
report  lists  all  of  the  asset  information  collected  in  a  smaller  area;  therefore,  the 
participant  must  read  every  line  of  text  in  the  situation  report.  The  participants  who 
had  the  map-based  display  were  able  to  scan  the  important  information  quickly, 
rather  than  having  to  read  through  an  entire  situation  report. 

Overconfidence  on  the  part  of  the  participants  using  the  text-based  display  is 
another  possible  explanation.  Participants  might  perceive  that  more  information  is 
provided  in  the  text-based  situation  report  compared  to  the  map-based  symbols.  In 
actuality,  the  same  amount  of  information  was  displayed  across  display  formats. 
The  text  display  may  appear  as  if  it  shows  more  data,  but  it  actually  uses  more 
working  memory  to  read  and  comprehend  the  information  than  is  required  by  the 
map  display  (i.e.,  scanning  symbols  on  a  grid). 
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Working  memory  is  the  part  of  the  memory  we  rely  on  to  plan  and  carry  out  specific 
behaviors  (Cowan  2009).  Since  the  map  is  using  less  working  memory,  the 
participants  are  able  to  make  decisions  more  efficiently  than  the  participants  using 
the  situation  report,  which  supports  the  results  from  the  experiment.  Not  only  did 
the  map  participants  score  more  hits,  they  completed  the  game  in  less  time  with  an 
average  time  of  162  s.  The  situation  report  participants  averaged 
258  s  to  complete  the  game. 

4.  Conclusion 


An  experiment  in  the  form  of  a  game  called  Shadow  Force  was  designed  and 
expanded  to  investigate  2  information  display  formats  and  2  types  of  assets.  The 
results  from  the  experiment  show  information  presented  using  the  map-based 
display  format  (symbols  on  a  grid)  increased  situational  awareness  compared  to  the 
text-based  situation  report.  The  map-based  display  resulted  in  a  greater  degree  of 
success  (time  and  number  of  hits)  in  completing  the  mission.  This  could  mean  that 
combat  effectiveness  would  be  improved  if  our  Soldiers  transitioned  to  a  more 
symbol-based  approach  to  information  displays,  in  accordance  with  NCW.  Shadow 
Force’s  data  could  drastically  change  the  way  our  Soldiers  view  information  from 
the  battlefield.  This  display  of  information  could  increase  information  supremacy 
and  accuracy  when  making  life-threatening  decisions,  and  thus  could  save  Soldiers’ 
lives  and  increase  combat  effectiveness. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


DARPA  Defense  Advanced  Research  Projects  Agency 

NCW  Network  Centric  Warfare 

UAV  unmanned  aerial  vehicle 
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